{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6f0dfc5c",
   "metadata": {},
   "source": [
    "## 5.7 梯度消失和梯度爆炸"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e01861e1",
   "metadata": {},
   "source": [
    "### 5.7.1 什么是梯度消失和梯度爆炸"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "471739fd",
   "metadata": {},
   "source": [
    "上一节我们讲了反向传播链式法则，通过链式法则，可以将误差从末层逐层向前传递，最终达到调整各层模型参数的目的。链式法则的形式如下：\n",
    "\n",
    "大家看到这个公式有没有发现什么问题？链式法则是一个连乘的形式，当模型层数浅的时候可能看不出来，随着层数的加深，梯度将以指数形式变化。\n",
    "\n",
    "当每一层的梯度都小于1的时候，随着网络层数加深，梯度将趋近于0，这就是梯度消失。相反，当每一层的梯度都大于1的时候，随着网络层数加深，梯度将趋近于正无穷，这就是梯度爆炸。梯度消失问题和梯度爆炸问题一般随着网络层数的增加会变得越来越明显，他们在本质原理上其实是一样的。\n",
    "\n",
    "不稳定梯度会威胁到我们优化算法的稳定性。 梯度爆炸发生，参数更新过大，破坏了模型的稳定收敛；梯度消失发生时，参数更新过小，则是更新时几乎不会移动，导致模型无法学习。\n",
    "\n",
    "无论是梯度趋近0还是正无穷，都会导致我们的模型训练失败。因此，解决梯度消失和梯度爆炸问题是深度学习的必修课。接下来我们就来实际看一下梯度消失和梯度爆炸出现的原因。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7781a77c",
   "metadata": {},
   "source": [
    "## 5.7.2 梯度消失"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af1f46f5",
   "metadata": {},
   "source": [
    "梯度消失是指当梯度传递到深层时，由于参数的初始值或激活函数的形式，梯度变得非常小，从而导致训练难以收敛。这种情况通常发生在使用 sigmoid 或 tanh 作为激活函数的情况下，因为这两个函数在输入较大时，梯度会变得非常小。\n",
    "\n",
    "下面代码绘制了sigmoid函数和它所对应的梯度函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b57cb6e6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAucUlEQVR4nO3deXhU5d3/8fd3JnsIhCxsCSFBw74TQBYVRVFEQFoRKFp3qo+gKLVq3VDbPm5tf/ZxaS1SVFBEsYqKijuykwCK7GEJCVvIQiBkn7l/f5wAIQYTYJIzM/m+ritX5p45mfkkDJ+c3GcTYwxKKaV8n8PuAEoppTxDC10ppfyEFrpSSvkJLXSllPITWuhKKeUnAux64ZiYGJOYmGjXyyullE9KS0vLMcbE1vSYbYWemJhIamqqXS+vlFI+SUQyTveYTrkopZSf0EJXSik/oYWulFJ+wrY59JqUl5eTlZVFSUmJ3VF8RkhICPHx8QQGBtodRSllM68q9KysLCIiIkhMTERE7I7j9Ywx5ObmkpWVRVJSkt1xlFI2q3XKRURmiUi2iPx0msdFRP4hIuki8qOI9DnbMCUlJURHR2uZ15GIEB0drX/RKKWAus2hzwau/IXHRwDJlR+TgVfOJZCW+ZnRn5dS6rhap1yMMUtEJPEXFhkDvGGs8/CuFJFIEWltjNnvqZBKKXUmjDGUuwxlLjel5S7KXG7KKio/XG5cbvPzD2OocBvcbutzbcsYY3AbMJWvZwy4jcFQ+dlUvR8M1vIYw7DOLenZNtLj37cn5tDjgMwq46zK+35W6CIyGWstnoSEBA+8dMO47bbbuO++++jSpUu9vcZVV13FW2+9RWRk5Cn3z5gxgyZNmvD73/++3l5bKW9QXOYiv6iMvGNlHC4qJ6+ojCPF5RwrreBYmYtjpRUUlVVQWGrdtu6voKjURWmFm9IK6/Px0vbWSz2IQMtmIV5b6HVmjHkVeBUgJSXFS3/cPzdz5sx6f41FixbV+2soZYeScheZeUXsKyjhQEExBwpKOXCkmAMFJRw8Ukp+URn5RWWUlLt/8XnCg5yEBQfQJDiA8GAnYUEBtIgIITTaSUiAk+BAB0FOB8EB1kdQ5UdwgNO67XQQHOggwOEgwCE4nYJThACH4HBU+yxCgLPKbYcDh4MTn51i3S9iTXuKYI2hyv0gCI7KZY5/rk+eKPS9QNsq4/jK+3zSsWPHuO6668jKysLlcvHoo4/yyiuv8Pzzz5OSksJrr73GM888Q2RkJD179iQ4OJgXX3yRm266idDQUNatW0d2djazZs3ijTfeYMWKFQwYMIDZs2cD8Pbbb/OXv/wFYwwjR47kmWeeAU6eCiEmJoY///nPvP7667Ro0YK2bdvSt29fG38iStVN9pESNu47Qnp2Ibtyj7E7x/rYf6TkZ2vLMU2CaNUshNbNQujSpilR4UFEhgUSFRZEZFgQUeFBNA8LpFloIOHBAYQGOnE4dHtRbTxR6AuBKSIyDxgAFHhi/vyJjzayad+Rcw5XVZc2TXl8VNdfXOazzz6jTZs2fPLJJwAUFBTwyivWdt59+/bx1FNPsXbtWiIiIrj00kvp2bPnia/Nz89nxYoVLFy4kNGjR7Ns2TJmzpxJv379WL9+PS1atOCBBx4gLS2N5s2bM3z4cD744AOuueaaE8+RlpbGvHnzWL9+PRUVFfTp00cLXXmdfYeLWbfnMBv3FbBx3xE27jtCTmHpiccjwwJJjA5nQPtoEqPDSYwJIy4ylJZNQ2jZNISgAD2msT7UWugi8jYwFIgRkSzgcSAQwBjzT2ARcBWQDhQBN9dX2IbQvXt3pk+fzgMPPMDVV1/NhRdeeOKx1atXc/HFFxMVFQXAuHHj2LZt24nHR40ahYjQvXt3WrZsSffu3QHo2rUru3fvJiMjg6FDhxIba50obdKkSSxZsuSUQv/+++8ZO3YsYWFhAIwePbq+v2WlarW/oJjvt+Wwalceq3blkpVfDECAQ0huGcHQjrF0bdOUrm2a0aFlEyLDgmxO3DjVZS+XibU8boC7PJaoUm1r0vWlQ4cOrF27lkWLFvHII48wbNiwOn9tcHAwAA6H48Tt4+OKigo9mlP5DLfbsD7rMF9vzuarLdls3m/9tRwVHkT/xChuGZxESmJzOraKIDjAaXNadZxXHSnqDfbt20dUVBTXX389kZGRp2wQ7devH9OmTSM/P5+IiAgWLFhwYi28Lvr378/dd99NTk4OzZs35+2332bq1KmnLHPRRRdx00038dBDD1FRUcFHH33E7373O499f0r9kq0HjvLB+r0sXL+PvYeLcTqEvu2a8+CITgztGEvHlhF67IMX00KvZsOGDdx///04HA4CAwN55ZVXTuwyGBcXxx//+Ef69+9PVFQUnTp1olmzZnV+7tatW/P0009zySWXnNgoOmbMmFOW6dOnD+PHj6dnz560aNGCfv36efT7U6q64jIXH6zfy5srMti0/whOhzDk/BimD+/AsE4taRamf1n6CjE27ayZkpJiql/gYvPmzXTu3NmWPHVVWFhIkyZNqKioYOzYsdxyyy2MHTvW1ky+8HNT3iczr4g3V2bwzppMCorL6dQqggn92jKyRxtiI4JrfwJlCxFJM8ak1PSYrqGfoRkzZvDll19SUlLC8OHDT9mgqZQv2Hu4mBe/3s67qVkY4MqurbhxUCL9EpvrdIqP00I/Q88//7zdEZQ6K9lHSnjxm3TmrbYO7P7NgATuuPg82kSG2pxMeYoWulJ+rsLl5vUVGfz9i22UlLsYlxLPlEuTidMi9zta6Er5sbSMPB7+709sOXCUizvE8sToriTGhNsdS9UTLXSl/FBJuYunP93C7OW7ad0shH9e34crurbSOXI/p4WulJ/ZfvAoU99ex5YDR7lpUCL3X9GR8GD9r94Y6AkV6lliYiI5OTkADBo06KyfZ/bs2ezbt89TsZQfMsbw9uo9jHpxKYeOlvKfm/sxY3RXLfNGRP+lz0JFRQUBAWf+o1u+fPlZv+bs2bPp1q0bbdq0OevnUP6rrMLNw//dwLtpWVyYHMNfr+tJi4gQu2OpBqaFXoOnnnqKOXPmEBsbe+L0tR9//DG9evVi6dKlTJw4kQ4dOvCnP/2JsrIyoqOjmTt3Li1btiQ3N5eJEyeyd+9eBg4cSNUDt5o0aUJhYSEAzz33HPPnz6e0tJSxY8fyxBNPsHv3bkaMGMGQIUNYvnw5cXFxfPjhh3zyySekpqYyadIkQkNDWbFiBaGhuoeCshQUlXPHnDRW7Mzl7mHJTBuWrKeabaS8e8rlPyNh3VzrtqvcGv/wjjUuK7LGPy2wxiUF1njTQmt8LNcab/3UGh89WKeXXLNmDQsWLOCHH37g008/perRrGVlZaSmpjJ9+nSGDBnCypUrWbduHRMmTODZZ58F4IknnmDIkCFs3LiRsWPHsmfPnp+9xuLFi9m+fTurV69m/fr1pKWlsWTJEgC2b9/OXXfdxcaNG4mMjGTBggVce+21pKSkMHfuXNavX69lrk7Yk1vEr15ZRmpGHn+7rif3Xd5By7wR0zX0apYtW8aYMWMICQkhJCSEUaNGnXhs/PjxJ25nZWUxfvx49u/fT1lZGUlJSQAsWbKE999/H4CRI0fSvHnzn73G4sWLWbx4Mb179was0wls376dhIQEkpKS6NWrFwB9+/Zl9+7d9fSdKl/3094CfjtrNW5jmHPrAAa0j7Y7krKZdxf6zZ+cvO0MPHUcFHbqOKTZqePw6FPHES3POU54+Mn9d6dOncp9993H6NGj+fbbb5kxY0adn8cYw0MPPfSzsyju3r37lNPuOp1OiouLzzm38j8b9xVw/WurCA8K4M1b+9M+tondkZQX8O4pFxsMHjyYjz76iJKSEgoLC/n4449rXK6goIC4uDgAXn/99RP3X3TRRbz11lsAfPrpp+Tn5//sa6+44gpmzZp1Yj597969ZGdn/2KuiIgIjh49elbfk/Ivm/YdYdLMVYQFOnn79gu0zNUJ3r2GboN+/foxevRoevToceKqQzWdInfGjBmMGzeO5s2bc+mll7Jr1y4AHn/8cSZOnEjXrl0ZNGgQCQkJP/va4cOHs3nzZgYOHAhYG0vnzJmD03n6CwXcdNNN3HHHHbpRtJGzynwloYFO3p58AQnRYXZHUl5ET59bg+OnyC0qKuKiiy7i1VdfpU+fPnbHOi1v+bmp+rU75xi/emU5wQEO5k2+gHbRegh/Y6Snzz1DkydPZtOmTZSUlHDjjTd6dZmrxuFwURm3zF6DMYa3btcyVzXTQq/B8TlwpbxBWYWbO+akkZVfzJzbBpCkJ9dSp+F1G0XtmgLyVfrz8m/GGB56fwMrd+bxzLXd6Z8UZXck5cW8qtBDQkLIzc3VkqojYwy5ubmEhOgh3v7q5W93sGBtFvcMS2Zs73i74ygv51VTLvHx8WRlZXHo0CG7o/iMkJAQ4uP1P7o/Wpaew/OLtzK6ZxumXZZsdxzlA7yq0AMDA08ccalUY3boaCnT3llP+5hwnv51dz2PuaoTryp0pRS43Yb75q/nSHE5b9zSn7Ag/W+q6sar5tCVUvDPJTv4fnsOj4/qSufWTe2Oo3yIFrpSXiQtI4+/Lt7GyB6tmdi/rd1xlI/RQlfKSxSXubj3nR+Iiwzlf3+l8+bqzOnknFJe4q+Lt7Inr4h5ky+gaUig3XGUD9I1dKW8wLo9+cxatotJAxK4QM9rrs6SFrpSNiurcPPAgh9p2TSEB0d0sjuO8mF1KnQRuVJEtopIuog8WMPjCSLyjYisE5EfReQqz0dVyj+9/G062w4W8uex3YjQqRZ1DmotdBFxAi8BI4AuwEQR6VJtsUeA+caY3sAE4GVPB1XKH209cJSXvknnml5tuLTTuV9VSzVudVlD7w+kG2N2GmPKgHnAmGrLGOD4DrPNgH2ei6iUfzLG8OiHP9EkOIDHRnW1O47yA3Up9Dggs8o4q/K+qmYA14tIFrAImFrTE4nIZBFJFZFUPV+Lauw+/ekAq3flMX14R6LCg+yOo/yApzaKTgRmG2PigauAN0XkZ89tjHnVGJNijEmJjY310Esr5XtKyl38ZdFmOrWKYEI/PYBIeUZdCn0vUPUdF195X1W3AvMBjDErgBAgxhMBlfJHry3dRVZ+MY9d3YUAp+5spjyjLu+kNUCyiCSJSBDWRs+F1ZbZAwwDEJHOWIWucypK1eDgkRJe+iadK7q2ZND5ut6jPKfWQjfGVABTgM+BzVh7s2wUkSdFZHTlYtOB20XkB+Bt4CajV6lQqkbPfraVCpfh4auq7yym1Lmp06H/xphFWBs7q973WJXbm4DBno2mlP/ZkFXAgrVZ3HHxeSREh9kdR/kZnbxTqgE9+/kWosKDuOuS8+yOovyQFrpSDWTVzly+357DnRefp0eEqnqhha5UAzDG8NfF22gREcz1F7SzO47yU1roSjWApek5rN6dx5RLzyc0yGl3HOWntNCVqmfGGJ7/fCtxkaGM14OIVD3SQleqnn25OZsfsgq4Z1gywQG6dq7qjxa6UvXI7Tb8dfFWkmLC+VWf6qdAUsqztNCVqkeLNx1ky4GjTLssWQ/xV/VO32FK1RNjDK98m0676DBGdm9tdxzVCGihK1VPlu/I5YesAn530Xm6dq4ahL7LlKonr3y7g9iIYJ07Vw1GC12pevBj1mGWpudw25AkQgJ1zxbVMLTQlaoHL3+zg6YhAfxmQILdUVQjooWulIelZxfy+aYD/HZgop6zRTUoLXSlPOxf3+0gyOngpsGJdkdRjYwWulIedPBICR+s38v4fm2JaRJsdxzVyGihK+VBc1ZmUOE23Dokye4oqhHSQlfKQ0rKXcxdtYdhnVrSLjrc7jiqEdJCV8pDPly/l7xjZdwyJNHuKKqR0kJXygOMMcxauptOrSIY2D7a7jiqkdJCV8oDVuzIZevBo9wyJAkRsTuOaqS00JXygFnLdhEdHsTonm3sjqIaMS10pc7RrpxjfLUlm0kDEvQwf2UrLXSlztHry3cT4BC9+LOynRa6UuegsLSC99KyuLpHG1o0DbE7jmrktNCVOgcfrNtLYWkFNwzUtXNlPy10pc6SMYY5KzPo0ropvdtG2h1HKS10pc5WWkY+Ww4c5YaB7XRXReUVtNCVOktzVmYQERzAmF66q6LyDlroSp2F3MJSFm04wK/7xhMWFGB3HKWAOha6iFwpIltFJF1EHjzNMteJyCYR2Sgib3k2plLeZX5qFmUuN5P0ikTKi9S6aiEiTuAl4HIgC1gjIguNMZuqLJMMPAQMNsbki0iL+gqslN1cbsNbqzO4oH0UyS0j7I6j1Al1WUPvD6QbY3YaY8qAecCYasvcDrxkjMkHMMZkezamUt5jybZDZOYV64FEyuvUpdDjgMwq46zK+6rqAHQQkWUislJErqzpiURksoikikjqoUOHzi6xUjabuyqDmCbBDO/Syu4oSp3CUxtFA4BkYCgwEfi3iERWX8gY86oxJsUYkxIbG+uhl1aq4RwoKOHrLdlclxJPUIDuU6C8S13ekXuBtlXG8ZX3VZUFLDTGlBtjdgHbsApeKb+yYG0WbgPXpbStfWGlGlhdCn0NkCwiSSISBEwAFlZb5gOstXNEJAZrCman52IqZT+32/DOmkwGto8mMUYvMae8T62FboypAKYAnwObgfnGmI0i8qSIjK5c7HMgV0Q2Ad8A9xtjcusrtFJ2WLkzlz15RUzor2vnyjvV6YgIY8wiYFG1+x6rctsA91V+KOWX5q3JpFloIFd01Y2hyjvpVh2l6iD/WBmf/XSAsb3j9CIWymtpoStVBx+s30uZy834fjrdoryXFrpStTDGMG91Jj3bRtK5dVO74yh1WlroStVifeZhth48ygRdO1deTgtdqVq8syaTsCAno3rqaXKVd9NCV+oXFJZWsPCHfVzdozVNgvU0ucq7aaEr9Qs++XEfRWUuxvfT0+Qq76eFrtQvmLcmk+QWTeiTEGl3FKVqpYWu1GlsPXCUdXsOM75fW71mqPIJWuhKncY7azIJdAq/6hNvdxSl6kQLXakalFa4eH9dFsO7tiIqPMjuOErViRa6UjVYvPEgh4vKdd9z5VO00JWqwTtrMomLDGXweTF2R1GqzrTQlapmT24RS9NzGN+vLQ6HbgxVvkMLXalq5qdm4hAYl6IbQ5Vv0UJXqooKl5t30zIZ2rEFrZuF2h1HqTOiha5UFd9tO8TBI6V6mlzlk7TQlapi3ppMYpoEc2mnFnZHUeqMaaErVSn7SAlfb8nm2r7xBDr1v4byPfquVarSe2uzcLmNTrcon6WFrhTWVYneWZPJgKQokmLC7Y6j1FnRQlcKWLkzj4zcIib017Vz5bu00JUC3lmzh4iQAEZ0a213FKXOmha6avQOF5Wx6KcDjO0dR0ig0+44Sp01LXTV6L2/di9lFW7dGKp8nha6atSMMby9eg8920bStU0zu+ModU600FWjlpaRz/bsQib112uGKt+nha4atbdW7aFJcABX99SNocr3aaGrRutwURkfb9jPNb3bEBYUYHccpc6ZFrpqtP67ztoY+pv+7eyOopRHaKGrRskYw1urrI2hXdo0tTuOUh5Rp0IXkStFZKuIpIvIg7+w3K9FxIhIiuciKuV5xzeG/kaPDFV+pNZCFxEn8BIwAugCTBSRLjUsFwHcA6zydEilPO2t1dbG0FE929gdRSmPqcsaen8g3Riz0xhTBswDxtSw3FPAM0CJB/Mp5XEFReV88qNuDFX+py6FHgdkVhlnVd53goj0AdoaYz75pScSkckikioiqYcOHTrjsEp5wvvrsiitcDNR9z1XfuacN4qKiAP4GzC9tmWNMa8aY1KMMSmxsbHn+tJKnTE9MlT5s7oU+l6g6paj+Mr7josAugHfishu4AJgoW4YVd4oLSOfbQd1Y6jyT3Up9DVAsogkiUgQMAFYePxBY0yBMSbGGJNojEkEVgKjjTGp9ZJYqXNwfGPo1T10Y6jyP7UWujGmApgCfA5sBuYbYzaKyJMiMrq+AyrlKfnHyk5sDA0P1o2hyv/U6V1tjFkELKp232OnWXboucdSyvPmrcmktMLNDRck2h1FqXqhR4qqRsHlNsxZmcHA9tF0bBVhdxyl6oUWumoUvtp8kL2Hi7lxkJ63RfkvLXTVKLy+YjdtmoVwWeeWdkdRqt5ooSu/l559lGXpuUy6oB0BTn3LK/+l727l915fnkFQgIMJes1Q5ee00JVfO1JSzoK1WYzq0YboJsF2x1GqXmmhK7/2floWRWUu3RiqGgUtdOW3XG7D7OW76dU2kh7xkXbHUareaaErv/XFpoPszi3i9gvb2x1FqQahha781r+/30nbqFCu6Kq7KqrGQQtd+aW0jHzSMvK5dXCS7qqoGg19pyu/NPP7nTQLDWRciu6qqBoPLXTldzJyj/HZxgNMGpCgZ1VUjYoWuvI7ry3dRYBDuGlQot1RlGpQWujKr+QfK2N+aibX9IqjRdMQu+Mo1aC00JVfeXNlBiXlbm6/SHdVVI2PFrryG4WlFby2dBfDOrWgQ0s957lqfLTQld94Y8VuCorLuXtYst1RlLKFFrryC8dKK5j5/S6GdoylZ9tIu+MoZQstdOUX5q7KIO9YGVMv1bVz1XhpoSufV1zm4tUlO7kwOYa+7ZrbHUcp22ihK5/31uo95BSW6dy5avS00JVPKyl38c/vdjCwfTT9EqM886SHM2Hfes88l1INSAtd+bS5q/Zw6GgpU4edf/ZPsvkj+OCuk+Nv/xfm/ebk+Ksn4b1bwZizfw2lGoAWuvJZR0rKefHr7Qw5P4ZB58Wc2Rdnrga3y7qdtxOyVkNFmTUecAeM/dfJZZ1BENwERKxx6iw4uPHcvwGlPEwLXfmsf367g/yich4c0enMvnD7l/Da5bB9sTUeOAWmrIGAIGvcugckXXhy+aEPwqgXrNulhfDFDEh7/ZzzK+Vpeio65ZP2FxTz2tJdXNOrDd3imtX+BeUlcDgDYjvCeZfA1X+HpIutxxzOur9wcBO4Z/3JtfVD22DHV9B/8pk9j1L1QNfQlU/6+xfbMAamD+9Yty949yaYcy1UlFrFm3ILBIWd3YuHRUFo5e6RP7wN3z4NRXln91xKeZCuoSufs+3gUd5Ly+KWwUm0jaqllI2x1qaHPmCVbkCwZ8MMewz63ghNYq1x9mZo0dmzr6FUHekauvI5z3y6hfDgAO665Bf2bHG7YNH9sOQ5a9ymN5w/zPNhRKB5onV743/h5YGw8zvPv45SdVCnQheRK0Vkq4iki8iDNTx+n4hsEpEfReQrEWnn+ahKwdLtOXy1JZv/GXo+zcODTr+gOKDkCJQeabjdDZOvgMtmQLvBDfN6SlVTa6GLiBN4CRgBdAEmikiXaoutA1KMMT2A94BnPR1UqdIKF499+BOJ0WHcPDix5oXydsHRg9aa8zWvwPA/ndyAWd+CwmDINHAGQEkBfDgFjuU2zGsrRd3W0PsD6caYncaYMmAeMKbqAsaYb4wxRZXDlUC8Z2MqBf9espOdOcd4ckw3QgJr2KPEVQ5vjoUPKw8Sctg4o7j/B9j0IWTr/uqq4dRlo2gckFllnAUM+IXlbwU+rekBEZkMTAZISEioY0SlIDOviP/7Op2R3VtzUYfYmhdyBsLIv0JE64YNV5Oki2Dajyf3hqkoO7mfu1L1xKOrMCJyPZACPFfT48aYV40xKcaYlNjY0/ynVKoaYwyPL9xIgEN49Orqs31Ya8PbKg8SOn8YtKxhGTscL/Od38GLfeHQVnvzKL9XlzX0vUDbKuP4yvtOISKXAQ8DFxtjSj0TTyn4YtNBvt6SzSMjO9OqWQ0Xfv7icSjItA4YcgY2fMDaNGkJ0ckQrisxqn7VpdDXAMkikoRV5BOA31RdQER6A/8CrjTGZHs8pWq0jpaU88RHm+jUKoIbByXWvNC1s6Cs0DvLHKBFJ7jhfeu22w15OyBGT/WrPK/WKRdjTAUwBfgc2AzMN8ZsFJEnRWR05WLPAU2Ad0VkvYgsrLfEqlF56uNN7C8o5s9juxPorPJ23fU9fHSPtb95WBRE+sg2me+egX9dDIf32J1E+aE6HSlqjFkELKp232NVbl/m4VxKsXjjAeanZnHXJef9/EpEe9Ngz0pr98AwD50HvSGk3GzNrfvKLyDlU8TYdI7nlJQUk5qaastrK++XU1jKFX9fQqtmIfz3fwYTFFC5dl51b5GyYxAUbl/Ic5WfATnbIVnXh1TdiUiaMSalpsf00H/ldYwxPLhgA0dLK/j7+F4ny3zLJ/BSP6sIwbfLHOCLR+GDO61fTEp5gJ6cS3mdd1Oz+HLzQR4Z2ZkOLSNOPhDZDmI6Qmikbdk8avT/WXPpvv6LSXkNXUNXXmXLgSM8vnAjF7SP4pbBSdadOenW51bdYNJ8CKnD+c99QUgzaNXdur32TVj/lr15lM/TQldeo6ConN+9mUZESAD/mNAbh0Mg/UtrmmVrjQcf+we3Gza+b52tUa9bqs6BTrkor+ByG+55Zx37Dhczb/IFtGhaeQBR4oUw9CFof4m9AeuTwwET3gYqz93udunVj9RZ0TV05RX+35fb+HbrIR4f1ZW+Cc1h3VxrY2FAMFz8Bwis4QhRfxIYAoGh1qXy5l4Ly/5hdyLlg7TQle0+++kA//d1OtelxDNpQAIc3AgLp0Dqf+yO1vAcTgiNgrBou5MoH6RTLspWa3bncc+8dfRsG8mTY7ohItbGz5s/g/h+dsdreM5A+PXMk+dwL8iCpnENd0535dN0DV3ZZvP+I9wyew1xkaHM+k1XQj66EzLXWA8mDLD3fOZ2Ol7e+RnwyiBY9oK9eZTPaKT/Y5TdMnKP8dtZq2kSHMCbtw0gOqgCstbAgR/tjuY9IhNgwJ3Q7Vd2J1E+QqdcVIPLPlLCDa+tpsLl5t0J7YhrGgyOULhzubVhUFlE4JKHTo5XvAzdfg0RLe3LpLyarqGrBpWZV8R1/1pBTmEpc8bFkfju5bD0b9aDWuanl78bvn4K1r5hdxLlxXQNXTWY7QePcv1rqyguc/HmrQPomhAJ2VOg+7V2R/N+zRPhd0sg6jxrXHoUgiN+8UtU46Nr6KpBrM88zLh/rSDYXcx3XRbSt3mxNaVw8R+sslK1i0m2NhSXHIFXh8J3z9qdSHkZLXRV777ecpBJ/15JREgA88Yn0HzHR7B7md2xfFdgKJx/ObQbbHcS5WV0ykXVG7fb8I+vt/PCl1sZH7uHe2+/lZZNQ+Ce9b51UQpv4wyEEU+fHKfOsi6a0XWsfZmUV9BCV/WioLice99Zz9dbsnkuaR3j9v8VCvpD035a5p7kdsOP71qF3uUaPQCpkdNCVx63dk8+985bR/Hhgzw1ZiDXplwKmztBfI0XWVHnwuGAGxdCeZFV5kf2w4EN0GG43cmUDbTQlceUlLv42xfbmPn9Tv4V+jKDW+UR1n8cOAOgx3V2x/NfzkBwVp4jfsWLsPrfMO1HiGhlby7V4LTQlUekZeQx490VbMpxMWFAEkOS7yS0LA9Et7s3qGGPQ6erT5b59i8hcYj/n61SAVro6hztPVzMc59tYe0P6/go+FGyhz5C8pWjge52R2ucAoKg3UDrdvYWmPtrGPYYXDjd3lyqQWihq7NSWFrBq19v4ptlK9hKArddNJBQ9/Uk99Rd6bxGi07w2w+hTW9rvG+ddQ3TTlfrBTT8lBa6OiN5x8p4ffluXl+xmyfL/8atwVspvHMdcTGRwDN2x1PVtR968vaqVyH9C0i+QgvdT2mhqzrJzCviva+W03zDLP5TNoYBXc6jc/eHaNbURbNoP7los78b8yLk7rDm042xrozUeRT0vcnuZMpDtNDVaZWUu/j6h3Q+TNvN57sq6Orcw4dBixl+za9oc4HuguhzHE6I7WDdLj0KUmUtvaIMtnwMHUfoSdJ8mBa6OkW5y83y9Bw+23iArzbs4Sv3bRwNuIyulz/OuJRLCQiaSJvQSLtjqnMV0hQmzT85Tv8C3rsZbvgvnHepdb4YRwAEhdmXUZ0xLXRFTmEpy3fk8u3WbC7a9DhBriIWynSGdY7jQOSjjOs+CEd8cuXSuvbmlzqMsDagHj8/TNp/4Nun4b5N1lGo5cUQEKJHono5LfRGKPvwMdZmHSUtI48WP/2b3seWcnfZDJqFBnJJi850inGQNvZyQgKdQG+746qG4HCcugE1cQgMuc8qc4BF98P+H+CO761x4SHrFA66cdWraKH7MbfbkHUwm4NbV7O6/Hw2HSqm3c53uLP8de4tfQmXM4x7YqOIiDifhSP60rVdS5wOPWRcAXF9rY/j2g+F6PNOjt+53ppr/+0H1njrZxDZFlp2bciUqhotdB/ndrnJPrSfzEIHewpcHMvaQIddbzLTcS3Lc8K4vOI7Xgh6mYdLn6G4eQfat+rCLse1zBvck07nJREcMMLub0H5guoXIRkw+eRGVWPgw7ugw5VwzUvWfe/dYp3it9dEa5yzHZq2gaDwhsvcCNWp0EXkSuAFwAnMNMY8Xe3xYOANoC+QC4w3xuz2bNTGwRhDaUkRR4rLOVLh5OiRAtw7l7AnMIndFdGU52UwbM8LzA8cw9KS84gr3MD8gMd4oOwBvnP3pLdjOzODlhEfM4zx/TrSI3IcOxz9+G+PSwmPiKx8lRvt/BaVP+j261PHt38Nxm3ddlVYl8wrzrPGFaXwYj8Y+hAMfcAaz70W+v8OOl8N5SWw4V1oN8j6K8DtguJ8a7pHp3TOSK2FLiJO4CXgciALWCMiC40xm6osdiuQb4w5X0QmYB1hMr4+AtcnYwxuA25XOcaAW5y43IaKY/mUGwdlzjAqXAby0il1hFIa0oIyl5uQ/akUBzXnaFgC5S5D9J7PKQiNJ7dJB4rLykneMZvM0K7sCOtBWUkxV2U8TVr4hawJGoC7tJCHsu/nXefVvO8aTHDxIZYF3smfym9mjutyWpPLipCpvFt+G++4L6VT6FEmsIvY0GIGJEWREDqANUV/4H+ShzOjXWfaRF5JcMA0ZpzynfWx5weqGgcRaN7u5NgZYBX8ccbAr2dCbCdrXFpolbq73BoXHoSFU2D0i1ahH86Af/SGa/5preHn7oC3xsOIZ+D8YdYvi6+egkFToU0vKNgLP86DbtdaOY4ehF3fQftLoEksFOVB9iZo1cPau6f0qLVMZFsICLY2+JYUQFiMld1VDq4yCAi1ti34kLqsofcH0o0xOwFEZB4wBqha6GPgRIe8B7woImKMMR7MCsD8NZmELv49Tc0RZoQ8gNsYHi75GwGmnEcC78dt4Onyp3Hh4A+O3+M2hlfcT3KEcKab+3Abw1x5hP0mmntc03AbwycB97PdxDOl/G4Avg26l3XmfO4tvwuAFcFTWOLqwQMVkwFIDb6D71z9ebTiFgB+DL6NBa4LeaLCWvPdHDyd113DebriN4AhPfgfpLpGMctEEBYo3E4qy4oS2Bfek4hAB0UBkcQ2j2R4i1ZEBrViZc5dDGp1If1b9aRZkCG99COmt07mTzEtCXA6gAn8/pSfygWe/jEr5TmBIadO2YRHw62LT46bxsE9P0JI5QFqIZEw4lmI72eNHU5o1c26H6xdKvetg7JCa5y/C756EuJSrELP3gjv3w43f2YVetYaeOs6uO1riO8LO76B+TfAHUuhVXfY8gksuBXuWg2xHa2/Fj64E+5eD1FJkPY6fDwNpv0EzeIgbTZ88RhMXWd9L6mz4Ju/wNS11i+M1f+GpX+3xoEhsOpfsPJla3mHA3LSIeb8evlR16XQ44DMKuMsYMDpljHGVIhIARAN5FRdSEQmA5MBEhISzipw8/AgjjVtS4k5RvfYZjgEynI746KCwS1icIhQlNMX43AyokUrBKEg+2LKnSGMa9EWh8Ch7JG4Appwa4skHAJ7D4yHwCimtUjGIcLufTcRHBLLH2I74hQh/eDdRIW14ekW3QlwOthx8E8kNWnDv2N6EOAUMnJm0ju8Je9HnUegw8H+I4u4IiyGqyJaEhLkoIRd3BESzpTAwMrvYgf3APec+K4upucp32X1E1vFndXPSimf4Aw4dQ0/LAoG/O7kuHkijJt9cty6B9y99uS43WB4+KB1GmGAthdYZdq0jTWO7we/XWhdkxUgrg/8aiY0a2uN2/SGkX+DJi0rn78XXPbEyQuxtOpu7fFz/KLc0cnQY4K1dg/QPMk6P87x149MsP46OD5dFNEa4vuf3OXz+NfVA6ltJVpErgWuNMbcVjm+ARhgjJlSZZmfKpfJqhzvqFwmp6bnBEhJSTGpqake+BaUUqrxEJE0Y0yNh2rXZYJoL9C2yji+8r4alxGRAKAZ1sZRpZRSDaQuhb4GSBaRJBEJAiYAC6sts5CTu05cC3xdH/PnSimlTq/WOfTKOfEpwOdYuy3OMsZsFJEngVRjzELgNeBNEUkH8rBKXymlVAOq037oxphFwKJq9z1W5XYJMM6z0ZRSSp0J39rJUiml1GlpoSullJ/QQldKKT+hha6UUn6i1gOL6u2FRQ4BGWf55TFUOwrVi3hrNs11Zrw1F3hvNs115s4mWztjTGxND9hW6OdCRFJPd6SU3bw1m+Y6M96aC7w3m+Y6c57OplMuSinlJ7TQlVLKT/hqob9qd4Bf4K3ZNNeZ8dZc4L3ZNNeZ82g2n5xDV0op9XO+uoaulFKqGi10pZTyEz5b6CLSS0RWish6EUkVkf52ZzpORKaKyBYR2Sgiz9qdpzoRmS4iRkRi7M4CICLPVf68fhSR/4pIpM15rhSRrSKSLiIP2pnlOBFpKyLfiMimyvfVPbV/VcMREaeIrBORj+3OUpWIRIrIe5Xvr80iMtDuTAAicm/lv+NPIvK2iIR44nl9ttCBZ4EnjDG9gMcqx7YTkUuwrrHa0xjTFXje5kinEJG2wHBgj91ZqvgC6GaM6QFsAx6yK0iVi6KPALoAE0Wki115qqgAphtjumBdRPYuL8l13D3AZrtD1OAF4DNjTCegJ16QUUTigLuBFGNMN6zTknvklOO+XOgGaFp5uxmwz8YsVd0JPG2MKQUwxmTbnKe6vwN/wPr5eQVjzGJjTEXlcCXWVbHscuKi6MaYMuD4RdFtZYzZb4xZW3n7KFYxecXFZkUkHhgJzLQ7S1Ui0gy4COt6DRhjyowxh20NdVIAEFp5hbcwPNRfvlzo04DnRCQTay3YtrW6ajoAF4rIKhH5TkT62R3oOBEZA+w1xvxgd5ZfcAvwqY2vX9NF0b2iOI8TkUSgN7DK5ijH/T+slQS3zTmqSwIOAf+pnA6aKSLhdocyxuzF6qw9wH6gwBiz2BPPXacLXNhFRL4EWtXw0MPAMOBeY8wCEbkO67fwZV6QKwCIwvqzuB8wX0TaN9Ql+WrJ9kes6ZYG90u5jDEfVi7zMNbUwtyGzOZLRKQJsACYZow54gV5rgayjTFpIjLU5jjVBQB9gKnGmFUi8gLwIPConaFEpDnWX31JwGHgXRG53hgz51yf26sL3Rhz2oIWkTew5u0A3qUB/9yrJdedwPuVBb5aRNxYJ+A5ZGc2EemO9Qb6QUTAmtZYKyL9jTEH7MpVJd9NwNXAMJuvR1uXi6LbQkQCscp8rjHmfbvzVBoMjBaRq4AQoKmIzDHGXG9zLrD+usoyxhz/S+Y9rEK322XALmPMIQAReR8YBJxzofvylMs+4OLK25cC223MUtUHwCUAItIBCMILzvRmjNlgjGlhjEk0xiRivdn7NESZ10ZErsT6k320MabI5jh1uSh6gxPrt/BrwGZjzN/sznOcMeYhY0x85XtqAtYF4r2hzKl8b2eKSMfKu4YBm2yMdNwe4AIRCav8dx2GhzbWevUaei1uB16o3KhQAky2Oc9xs4BZIvITUAbcaPMapy94EQgGvqj862GlMeYOO4Kc7qLodmSpZjBwA7BBRNZX3vfHyuv9qtObCsyt/OW8E7jZ5jxUTv+8B6zFmmJch4dOAaCH/iullJ/w5SkXpZRSVWihK6WUn9BCV0opP6GFrpRSfkILXSml/IQWulJK+QktdKWU8hP/H1KbL6u//WYVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "x = torch.arange(-8.0, 8.0, 0.1, requires_grad=True)\n",
    "y = torch.sigmoid(x)\n",
    "y.backward(torch.ones_like(x))\n",
    "\n",
    "plt.plot(x.detach().numpy(), y.detach().numpy(), label = 'sigmoid')\n",
    "plt.plot(x.detach().numpy(), x.grad.numpy() ,linestyle=':', label = 'gradient')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c358874",
   "metadata": {},
   "source": [
    "可以看出，当sigmoid函数的输入很大或是很小时，它的梯度都是一个远远小于1的数，非常趋近于0。当反向传播通过许多层时，除非每一层的sigmoid函数的输入都恰好接近于零，否则整个乘积的梯度可能会消失。 当我们的网络有很多层时，除非我们很小心，否则在某一层可能就会切断梯度。 因此，现在大家更愿意选择更稳定的ReLU系列函数作为激活函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6eed5220",
   "metadata": {},
   "source": [
    "## 5.7.3 梯度爆炸"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cc164b7",
   "metadata": {},
   "source": [
    "与之相反的则是梯度爆炸问题。梯度爆炸是指当梯度传递到深层时，由于参数的初始值或激活函数的形式，梯度变得非常大，从而导致训练难以收敛。为了更直观的看到这个问题，我们用代码生成了100个高斯随机矩阵，并将这些矩阵与一个矩阵相乘，这个矩阵相当于模型的初始参数矩阵。我们设置方差为1，看一下运行结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "1147274a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "初始参数矩阵 tensor([[ 1.5515, -0.5073, -2.4602, -1.9177,  0.4678],\n",
      "        [ 0.2328, -0.6902,  1.4514, -2.1545, -1.1679],\n",
      "        [ 0.5522, -0.4998,  0.9615,  0.0644, -0.0136],\n",
      "        [ 0.9784,  0.1601, -0.5344,  0.7700,  0.7958],\n",
      "        [ 0.5089, -1.4568,  0.8654, -1.0948,  1.3430]])\n",
      "计算后矩阵 tensor([[-1.0448e+29,  7.7966e+28,  9.1455e+28,  8.5349e+27, -2.6260e+29],\n",
      "        [ 4.9579e+28, -3.6996e+28, -4.3397e+28, -4.0499e+27,  1.2461e+29],\n",
      "        [ 3.3363e+28, -2.4895e+28, -2.9203e+28, -2.7253e+27,  8.3852e+28],\n",
      "        [-3.9100e+28,  2.9176e+28,  3.4225e+28,  3.1940e+27, -9.8272e+28],\n",
      "        [ 6.7301e+28, -5.0220e+28, -5.8909e+28, -5.4976e+27,  1.6915e+29]])\n"
     ]
    }
   ],
   "source": [
    "Mat = torch.normal(0, 1, size=(5,5))\n",
    "print('初始参数矩阵',Mat)\n",
    "for i in range(100):\n",
    "    Mat = torch.mm(Mat, torch.normal(0, 1, size=(5, 5)))\n",
    "\n",
    "print('计算后矩阵', Mat)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb5a8a1e",
   "metadata": {},
   "source": [
    "可以看到在经过100次乘法运算后，矩阵内的值发生了爆炸性增长，这就是梯度爆炸。这种情况其实是由于我们的参数初始化方法所导致的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "844a8495",
   "metadata": {},
   "source": [
    "## 5.7.4 解决方法"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b03c0ff7",
   "metadata": {},
   "source": [
    "解决梯度消失和梯度爆炸问题的方法很多，这里讲两种常见的方法，梯度裁剪和使用Relu函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a932c25",
   "metadata": {},
   "source": [
    "**梯度裁剪（正则化）**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe305f60",
   "metadata": {},
   "source": [
    "梯度裁剪主要是针对梯度爆炸提出。其思想也比较简单，训练时候设置一个阈值，梯度更新的时候，如果梯度超过阈值，那么就将梯度强制限制在该范围内，这时可以防止梯度爆炸。\n",
    "权重正则化（weithts regularization)也可以解决梯度爆炸的问题，其思想就是我们常见的正则方式。\n",
    "$$ Loss = (y-w^Tx)^2 + \\alpha \\left \\| w \\right \\|^2 $$ \n",
    "α是正则化系数。如果发生梯度爆炸，$\\left \\| w \\right \\|$的平方会变得非常大，这样就可以一定程度避免梯度爆炸。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd3dcb94",
   "metadata": {},
   "source": [
    "**采用Relu激活函数**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1c89b6a",
   "metadata": {},
   "source": [
    "ReLU函数在AlexNet网络中最先提出。是一种常用的神经网络中的激活函数，它可以将输入信号的值转化为 0 或正数。ReLU 函数可以被表示为 $f(x)=max(0,x)$。由于relu激活函数的导数不是1就是0，因此不存在梯度消失问题，不同层之间的梯度基本保持一致。\n",
    "\n",
    "而relu的缺点则是，负数部分恒为0，所以存在一定‘死区’，会导致一些神经元无法被激活，可以通过LeakyRelu等函数来改善死区的问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "438bc467",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAmqklEQVR4nO3dd3xUZdr/8c9NCqm0EHpJ1AAqQghRQZFdsTdc1wII7qP7UxTByupaV1m3uOKqWPdhdy3PEkAE1BUXxYKLBdEUOkgvoSUkQBrp9++PM4GgQCZDJmeS+b5fL17OlTkzc2USr5zcOed8jbUWEREJXC3cbkBERI5Pg1pEJMBpUIuIBDgNahGRAKdBLSIS4EL98aTt27e3CQkJ/nhqEZFmKSMjY6+1Nv5o9/llUCckJJCenu6PpxYRaZaMMVuPdZ+WPkREApwGtYhIgNOgFhEJcF6tURtj7gNuBSywArjFWltanxeqqKggOzub0tJ6PSyoRURE0K1bN8LCwtxuRURcVOegNsZ0Be4GTrPWHjTGzAJGAm/W54Wys7OJjY0lISEBY4xPzQYTay15eXlkZ2eTmJjodjsi4iJvlz5CgUhjTCgQBeys7wuVlpYSFxenIe0lYwxxcXH6DURE6h7U1todwLPANmAXcMBau+DH2xljxhpj0o0x6bm5uUd9Lg3p+tH7JSLgxaA2xrQFrgYSgS5AtDFmzI+3s9ZOtdamWmtT4+OPesy2iEiztWRTHv/8ajP+uHS0N0sfFwKbrbW51toKYC5wToN3EiBiYmLcbkFEmpicwlImzMhi2rdbOVhR1eDP782g3gYMMsZEGed38QuANQ3eSSOy1lJdXe12GyLSDFRWVXPPjKUUllbw2pgUosIb/oRvb9aolwCzgUycQ/NaAFMbvBM/27JlC7179+ZXv/oVffv25amnnuLMM8+kX79+PPHEEz/Z/osvvuDKK688VE+YMIE333yzETsWkabg+U/XsXhTHk9d3Zc+nVr55TW8Gv3W2ieAn04zH036YBWrdxY01NMBcFqXVjxx1enH3Wb9+vW89dZbFBQUMHv2bL777justQwfPpxFixYxdOjQBu1JRJq3z9fu4ZWFGxmR2p3rU7v77XWC6szEnj17MmjQIBYsWMCCBQsYMGAAKSkprF27lvXr17vdnog0Idn7Srjv7WWc2rkVk64+/k7iifLL1fPqUteer79ER0cDzhr1ww8/zO23337MbUNDQ49Yx9bxzCJSo6yyivFpmVRXW14bnUJEWIhfXy+o9qhrXHLJJbz++usUFRUBsGPHDnJyco7YpmfPnqxevZqysjL279/PZ5995karIhKA/vjhGpZlH2Dy9f1JaB/t99dzZY/abRdffDFr1qxh8ODBgHNI3rRp0+jQocOhbbp3784NN9xA3759SUxMZMCAAW61KyIB5P2lO/i/xVu57bxELu3bqVFe0/jj4OzU1FT74+CANWvWcOqppzb4azV3et9EAseGnEKGv/w1p3VuxYyxgwgLabhFCWNMhrU29Wj3BeXSh4hIfZWUVzJuWiaRYSG8fGNKgw7pugTl0oeISH1Ya3lk7go25Bbxr1+fTafWEY36+tqjFhGpw/TvtvHe0p3cd2EvhiS1b/TX16AWETmOFdkHmPTv1QztFc+E809xpQcNahGRYzhQUsG4tAzax4TzwohkWrRw59LDWqMWETmK6mrLxHeWsqeglLdvH0y76HDXetEetY8SEhLYu3cvAOec4/tVX99880127qx3YI6I+NnULzfx6ZocHrn8VFJ6tHW1Fw3qWiorK3163DfffOPza2pQiwSebzflMfnjH7jijM7cfE6C2+0E19LHU089xbRp04iPj6d79+4MHDiQefPmkZyczFdffcWoUaPo1asXf/jDHygvLycuLo60tDQ6duxIXl4eo0aNYseOHQwePPiIFIeYmJhDp6NPnjyZWbNmUVZWxjXXXMOkSZPYsmULl112GUOGDOGbb76ha9euvP/++3z44Yekp6czevRoIiMjWbx4MZGRkW69PSKCEwJw14wseraL4ulrzwiISDz39qjfuAKy0pzbVRVOvextpy4vceqVc5y69IBTr/63UxfnOfUP8526cE+dL/f9998zZ84cli1bxvz586l95mR5eTnp6elMnDiRIUOG8O2335KVlcXIkSN55plnAJg0aRJDhgxh1apVXHPNNWzbtu0nr7FgwQLWr1/Pd999x9KlS8nIyGDRokWAc4nV8ePHs2rVKtq0acOcOXO47rrrSE1NJS0tjaVLl2pIi7issqqau2dkUVhawatjUoiNCHO7JSCI9qi//vprrr76aiIiIoiIiOCqq646dN+IESMO3c7OzmbEiBHs2rWL8vJyEhMTAVi0aBFz584F4IorrqBt25+uWdW+fCpAUVER69evp0ePHiQmJpKcnAzAwIED2bJli58+UxHx1XOfrOPbTfn89fr+fgsB8EWdg9oY0xt4u9aHTgJ+Z6194YRe+ZYPD98OCTuyDo86so5ofWQdHXdkHdvxhFqpufwpwF133cX999/P8OHD+eKLL3jyySe9fp5jXT51y5YttGzZ8lAdEhLCwYMHT6hnEWlYn63Zw6tfbGTUWd25dmA3t9s5gjdRXD9Ya5OttcnAQKAEeNffjTW0c889lw8++IDS0lKKioqYN2/eUbc7cOAAXbt2BeCtt9469PGhQ4cyffp0AObPn8++fft+8lhvLp/6Y7GxsRQWFvr0OYlIw9ieX8L9s5ZxuhdJUW6o79LHBcBGa+1WfzTjT2eeeSbDhw+nX79+dOzYkTPOOIPWrVv/ZLsnn3yS66+/nrZt2zJs2DA2b94MwBNPPMGoUaM4/fTTOeecc+jRo8dPHnusy6eGhBz7ouI333wzd9xxh/6YKOKSssoqxk/PpNpaXm2EEABf1Osyp8aY14FMa+3LR7lvLDAWoEePHgO3bj1ylgfC5TqLioqIiYmhpKSEoUOHMnXqVFJSUlztqS6B8L6JNGePv7eSf327lf+9aSCXnN4415c+mga5zKkxJhwYDrxztPuttVOttanW2tT4+HjfOvWzsWPHkpycTEpKCtdee23AD2kR8a/3l+7gX99uZezQk1wd0nWpz9LHZTh703UfCxegataYRUTW7ynk4bkrODOhLQ9c0tvtdo6rPsdRjwJmnMiL+SNNpjnT+yXiH8VllYxLyyQqvPFDAHzhVXfGmGjgImCury8UERFBXl6eho+XrLXk5eUREdG4FygXae6stTzy7go25RYxZeQAOrYK/P/HvFr6sNYWA3En8kLdunUjOzub3NzcE3maoBIREUG3boF1PKdIU5e2ZBvvL93JxIt6ce4pjR8C4ItGOzMxLCzs0Fl+IiJuWJ69n99/sJqf945nvEshAL4I7IUZEZEGsr+knHHTMmkfE87zN7gXAuCLoLnWh4gEr+pqy8RZy8gpLGXW7YNp62IIgC+0Ry0izd7fFm3ks7U5PHbFaQxwOQTAFxrUItKsLd6Yx7Mf/8CV/Trzq8E93W7HJxrUItJs5RQ4IQAJ7aN5+tp+AREC4AutUYtIs1RZVc1dM7IoKqsg7daziWnZdMdd0+1cROQ4/vrJOpZszue5G/rTu1Os2+2cEC19iEiz8+nqPbz2xUZGndWDX6Y0/ZPGNKhFpFlxQgCWekIATnO7nQahQS0izUZpRRXj0jKwwGujBwZkCIAvtEYtIs3GU/NWs3JHAX//VSo94qLcbqfBaI9aRJqF97J2kLZkG7cPPYmLTjuxwOtAo0EtIk1eTQjAWQnt+E2AhwD4QoNaRJq04rJK7piWQXTLEF66cUDAhwD4QmvUItJkWWt5aO4KNu8tZtqtZzeJEABfeJvw0sYYM9sYs9YYs8YYM9jfjYmI1GXat1v5YNlOJl7cm3NObhohAL7wdo96CvCRtfY6Txp58/lzqog0Scu27+f381Zzfu94xv3sZLfb8as6B7UxpjUwFLgZwFpbDpT7ty0RkWPbX1LOnWmZdIiN4PkRTSsEwBfeLH0kArnAG8aYLGPMPzxht0cwxow1xqQbY9KViygi/lJdbbnfEwLwyugU2kQ1rRAAX3gzqEOBFOA1a+0AoBh46McbWWunWmtTrbWp8fHxDdymiIjjtf9u5PO1OTx+5Wkkd2/jdjuNwptBnQ1kW2uXeOrZOINbRKRRfbNxL39d8ANX9e/CTYOaZgiAL+oc1Nba3cB2Y0zNUeQXAKv92pWIyI/sKSjl7hlZJLaP5s+/PKPJhgD4wtujPu4C0jxHfGwCbvFfSyIiR6oJASguq2L6bYOadAiAL7z6bK21S4FU/7YiInJ0zy5Yx3eb83lhRDK9OjbtEABfNL9zLUWkWflk9R7+9t+NjD67B78Y0NXtdlyhQS0iAWtbXgkTZy2lb9dWPH5l8wgB8IUGtYgEpNKKKu6cngE0rxAAXwTXiryINBm/rxUC0L1dcF+1QnvUIhJw3s3KZvqSbdzxs5ObXQiALzSoRSSgrNtTyCNzV3JWYjt+c3Evt9sJCBrUIhIwig6FAITy8qgBhDbDEABf6F0QkYBgreXhuSvYsreYl0YNoEMzDQHwhQa1iASEf9UKARh8cpzb7QQUDWoRcd3S7ft5at5qhvXp0OxDAHyhQS0irtpXXM54TwjAczf0b/YhAL7QcdQi4prqast9s5aSW1jG7HGDgyIEwBfaoxYR17z234188UMuj191Gv26tXG7nYClQS0irvhmgxMCcHVyF8ac3cPtdgKaBrWINLo9BaXcPTOLk+Jj+NM1wRUC4AutUYtIo6qoqmbC9EyKy6qYcVsK0UEWAuALr94hY8wWoBCoAiqttQoREBGfPPvxD3y/ZR9TRiaTFIQhAL6oz4+y8621e/3WiYg0ewtW7eZ/F21izKAeXJ0cnCEAvtAatYg0im15JUx8Zxn9urUO6hAAX3g7qC2wwBiTYYwZe7QNjDFjjTHpxpj03NzchutQRJq80ooqxqVl0MIYXrkxhZahwRsC4AtvB/UQa20KcBkw3hgz9McbWGunWmtTrbWp8fHxDdqkiDRtkz5YzaqdBTx3Q/+gDwHwhVeD2lq7w/PfHOBd4Cx/NiUizcecjGxmfLeNcT8/mQtOVQiAL+oc1MaYaGNMbM1t4GJgpb8bE5Gmb+3uAh59bwVnJ7Zj4kUKAfCVN0d9dATe9RyQHgpMt9Z+5NeuRKTJKyqr5M60TGIjwnjpRoUAnIg6B7W1dhPQvxF6EZFmwlrLb+csZ8veYqbfNogOsQoBOBH6ESciDe7/Fm/lw+W7eOCSPgw6SSEAJ0qDWkQaVNa2ffzhw9VceGoHbh96ktvtNAsa1CLSYGpCADq2iuCv1ycrBKCB6GooItIgqqst9769lL1F5cweN5jWUWFut9RsaI9aRBrEKws38N91ufxOIQANToNaRE7Y1xv28tyn6/hFchdGKwSgwWlQi8gJ2X2glHtmZnFKfAx/VAiAX2iNWkR8VlFVzV0zMikpr2LmWIUA+IveVRHx2WRPCMCLowZwSgeFAPiLlj5ExCcfrdzN1EWbuGlQT4b37+J2O82aBrWI1NvWvGIe8IQAPHblqW630+xpUItIvZRWVDFuWiYtWigEoLFojVpE6mXSB6tYvauA129OVQhAI9EetYh4bXZGNjO+2874809mWB+FADQWDWoR8cra3QU89t4KBp8Ux30XKgSgMWlQi0idCksrGDfNCQGYMipZIQCNzOt32xgTYozJMsbM82dDIhJYrLU8NGcF2/JLeHnUAIUAuKA+PxbvAdb4qxERCUxvfrOFD1fs4oFLenO2QgBc4dWgNsZ0A64A/uHfdkQkkGRu28ef/rOGC0/tyNjzFALgFm/3qF8AHgSqj7WBMWasMSbdGJOem5vbEL2JiIvyi8uZkJZJp9YR/PX6/goBcFGdg9oYcyWQY63NON521tqp1tpUa21qfHx8gzUoIo3vUAhAcTmvjR6oEACXebNHfS4w3BizBZgJDDPGTPNrVyLiqpcXbmDRulyevOp0+nZt7XY7Qa/OQW2tfdha281amwCMBD631o7xe2ci4oqv1u/l+U/Xcc2Arow6q7vb7Qg6jlpEatl14CB3z8wiqUMMf7ymr0IAAkS9rvVhrf0C+MIvnYiIqyqqqpkwPYvSiipeHT2QqHBdCihQ6CshIgA889FaMrbWhADEuN2O1KKlDxHho5W7+PuXm/mfwQoBCEQa1CJBbsveYh54Zzn9u7fhkSsUAhCINKhFglhpRRXj0mpCAAYoBCBAaY1aJIg98f4q1uwq4I2bz6RbW4UABCrtUYsEqXfSt/N2+nYmnH8K5/fp4HY7chwa1CJBaM2uAh5/fyXnnBzHfRcpBCDQaVCLBJnC0gruTMukVUQYU0YOIEQXWwp4WqMWCSLWWn47Zznb8kuYcdsg4mNbut2SeEF71CJB5I2vt/CfFbv57aW9OSuxndvtiJc0qEWCRMZWJwTgotM6cptCAJoUDWqRIJBfXM6E6Zl0bhPBs9f318WWmhitUYs0c1XVlntmZpFXXM7ccefQOlIhAE2N9qhFmrmXP9/Al+v3Mmm4QgCaKg1qkWbsy/W5vPDZOn6Z0pWRZyoEoKnSoBZppnYdOMg9M5fSq0Msf/iFQgCaMm/CbSOMMd8ZY5YZY1YZYyY1RmMi4ruKqmrGp2VSVlHFq2NSFALQxHnz1SsDhllri4wxYcBXxpj51tpv/dybiPjo6flrydy2n5dGDeDkeIUANHV1DmprrQWKPGWY55/1Z1Mi4rv5K3bxz682c/M5CVylEIBmwas1amNMiDFmKZADfGKtXXKUbcYaY9KNMem5ubkN3KaIeGPz3mIenL2c5O5teORyhQA0F14NamttlbU2GegGnGWM6XuUbaZaa1Ottanx8fEN3KaI1KW0oopx0zIICTG8MjqF8FAdK9Bc1Osraa3dDywELvVLNyLis9+9v5If9hTywohkuraJdLsdaUDeHPURb4xp47kdCVwErPVzXyJSD7O+386s9GzuOv8Uft5bIQDNjTdHfXQG3jLGhOAM9lnW2nn+bUtEvLV6pxMCcO4pcdxzoUIAmiNvjvpYDgxohF5EpJ4KSiu4My2DNlEKAWjOdBS8SBNlreW3s5ezfd9BZo4dRPsYhQA0V/qzsEgT9frXW5i/cjcPXdqHMxMUAtCcaVCLNEEZW/P583/WcPFpHbn1vES32xE/06AWaWLyisoYn5ZF17aRTFYIQFDQGrVIE1JVbbn37aXkl5Tz7p0KAQgW2qMWaUJe/Gw9X67fy++Hn87pXRQCECw0qEWaiEXrcnnx8/Vcm9KNEQoBCCoa1CJNwM79B7lnZha9OyoEIBhpUIsEuPLKaiZMz6SiyvLq6BQiw0Pcbkkamf6YKBLgakIAXrkxhZMUAhCUtEctEsD+s2IXr3+9mVvOTeCKfp3dbkdcokEtEqA25Rbx4OzlDOjRhocvUwhAMNOgFglAB8uruDMtk7AQw8s3KgQg2GmNWiQA1YQAvHHzmQoBEO1RiwSaWd9v552MbO4alqQQAAE0qEUCyqqdB3j8/ZUMOaU991yQ5HY7EiC8ieLqboxZaIxZbYxZZYy5pzEaEwk2TghAJm2jwpkyMlkhAHKIN2vUlcBEa22mMSYWyDDGfGKtXe3n3kSChrWWB95Zxg5PCECcQgCkljr3qK21u6y1mZ7bhcAaoKu/GxMJJv/8ajMfr9rDQ5f1IVUhAPIj9VqjNsYk4OQnLjnKfWONMenGmPTc3NwGak+k+Uvfks/T89dyyekd+X9DFAIgP+X1oDbGxABzgHuttQU/vt9aO9Vam2qtTY2Pj2/IHkWarb1FZUyYrhAAOT6vjqM2xoThDOk0a+1c/7YkEhyqqi33zlzKvpJy5t55Dq0iFAIgR1fnoDbOj/h/Amustc/5vyWR4DDls/V8tWEvz1zbTyEAclzeLH2cC9wEDDPGLPX8u9zPfYk0a1/8kMNLn6/n+oHduEEhAFKHOveorbVfAVo4E2kgO/cf5L63l9K7Yyy/v7qv2+1IE6AzE0UaUXllNeMVAiD1pIsyiTSiP89fQ9a2/bw6WiEA4j3tUYs0kg+X7+KNr7fw63MTufwMhQCI9zSoRRrBxtwiHpy9jJQebXjosj5utyNNjAa1iJ8dLK/izmmZtAwLUQiA+ERr1CJ+ZK3lsfdWsi6nkLduOYsuCgEQH+hHu4gfzUrfzpzMbO4elsTQXrq0gvhGg1rET5wQgFWcl9SeuxUCICdAg1rEDw4cdEIA2kWF88IIhQDIidEatUgDqx0C8PbtCgGQE6c9apEG9o8vN7Ng9R4evvxUBvZUCICcOA1qkQb0/ZZ8nv5oLZf17cSvz01wux1pJjSoRRqIEwKQSY92UTxzXT+FAEiD0aAWaQBV1ZZ7Zmaxv6SCV0enEKsQAGlA+mOiSAOY8uk6vt6QxzPX9ePUzq3cbkeaGe1Ri5yghT/k8OLnG7ghtRs3pCoEQBpenYPaGPO6MSbHGLOyMRoSaUp2eEIA+nRSCID4jzd71G8Cl/q5D5Emp7yymjvTMqmqsrw2ZiARYQoBEP+oc1BbaxcB+Y3Qi4j/LXgMqquc24tfgZcGHr7vq+fh1cGH6/9Ohr+dd7he+Cf4+7BD5ZK/382je+5j8vX9SGwfDR8/Cm9eeXj7/zwI//rl4XrefTB95OH6/Qnw9pjD9dzbYfavD9ezf+18rMbbY5zH1Jg+0nnOGv/6pfOaNd680umpxuuXwqdPHq7/Psz5nGr87Tznc67x6mDnPanx0kDnPQPnPXxxAHz3d6euOOjUGW86dWmBU2elOXVJvlMvn+XUhbudetW7Tr1/u1OvmefUeRudet0Cp85Z69QbP3fq3SucevOXTr0jw6m3LXHqbUucekeGU2/+0ql3r3DqjZ87dc5ap163wKnzNjr1mnlOvX+7U69616kLdzv18llOXeIZjVlpTl1agD802B8TjTFjgbEAPXr0aKinFWlYq96HqDgYch+06gJdUg7f17o7dBlwuG7THbok16p7QOf+AMxbvpOvs8O5pusZXNrXEwLQNgFs9eHt250EobXOSmx3MkTUShuPOwXKiw7X7ZMO/xABaN8bWtTaS4/vA+G1UmE69IGo9ofrjqdDq66H605nOD3Vrtv0PFx37u98TjW6JDuf86F6gPOeHKpTnPcMAANdB0JsJ0/ZwqljOjp1ixBP3eHIOtrTb0i4U9f0H9rSU8d56ginjmzr1OFRTh3RxlNHe2rP+xke69QtY526pacO99QRrT11tKdu46mjnDqyrVOHRjh1VJyn9nz9oto7dUi4U0d76pqvT0yHI+sGZqy1dW9kTAIwz1rr1SJcamqqTU9PP8HWRPxg40LP/7QpdW97rKfILWL4S1/Rp3MrZo4dRFiI/iYvJ84Yk2GtTT3afTo8T4LLyeef0MNLyisZNy3DEwIwQENaGoW+yyR4VByErYsPryvWk7WWx95dyfqcIqaMTKZza4UASOPw5vC8GcBioLcxJtsY8//835aIH+Rvgjcuhc3/9enhM7/fztysHdx7QS/OS1IIgDSeOpc+rLWjGqMREb9r0wNuehc6nF7vh67ccYAn/r2Kob3iuWvYKX5oTuTYtEYtwaNlLJw8rO7tfqQmBCAu2gkBaKEQAGlkWqOW4LFvi3M8bVWl1w+x1vKbd5axc/9BXr4xhXbR4f7rT+QYNKgleKx4B966EmxV3dt6/P3LTXyyeg+PXH4qA3u29WNzIsempQ8JHsmjofugI09COY7vNufzl49+4PIzOnGLQgDERRrUEjxadal1Zt3x5RYeDgH4y7UKARB3aelDgseWryE7o87Nqqotd8/IoqC0gtfGKARA3KdBLcHjk8dh4R/r3Oz5T9axeFMef/jFGfTppBAAcZ+WPiR4XDP1yIsmHcXCH3J4eeEGRqR257qB3RqpMZHj06CW4NH++CeqZO8r4b63l3Jq51ZMurr+J8WI+IuWPiQ4VFXC8nec08iPoqyyivHTs5wQgNEpCgGQgKJBLcGhJA/m3gobPjvq3X/6cA3Ltu9n8vX9SWgf3cjNiRyflj4kOETFwYR0iGz3k7v+vWwnby3eym3nJXJp304uNCdyfBrUEhxCQp0ElR/ZkFPEQ3OWk9qzLQ9e2seFxkTqpqUPCQ55G2HpDCgrPPShkvJK7kzLIDIshJdvTFEIgAQsfWdKcNi8CN6749Cgttby6KEQgAF0ah3hcoMix6alDwkO/UZA4tBD4aszvtvOu1k7uP+iXgxJal/Hg0Xc5dUetTHmUmPMD8aYDcaYh/zdlEiDC4+CuJOhRQgrsg/wpCcEYML5CgGQwOdNFFcI8ApwGXAaMMoYc5q/GxNpUOs/gVXvcaCkgjunZ9A+RiEA0nR4s/RxFrDBWrsJwBgzE7gaWN3QzUx4fhq/KXyalyLGsTy0L6dUbeS3B5/j+YgJrA49lT6VPzCx9EUmR97LupAkTq9czb2lr/DnyN+wKSSR5MpljC+dylORD7EtpDuplZncXvpPnoh6lJ0tunB2xffcWvYmj0Y9QU6LDgyp+Ib/KUvjt1FPkd+iHT+r+JIxZTOZGPVnClq04sLyhYwon8090c9QYqK5pPwTrit/jwnRz1FmWnJF+Uf8ovwD7oieQpUJ5eryeVxR/hFjY14G4Nqy97iwYiHjYqYAMKJsDudWLObumGcBuLHsbc6szOS+6L8A8KvS6ZxRtYoHop3rUfy69C2SqjbxcPQkAMaWvk736mwej/odAHeWTiW+ei+Toh4B4O6DrxJri/hj1IMA3H/wJcIo5y+REwF48ODzVNOCZyPvAeCRksmUmEheiJwAwO9K/ky+acvLkXcA8PuSP7CzRSf+FnErAH8sfpLNIT35R8QtADxT/BirQ/rwZsQYAP5a/BBZof2Z1tJJb5tS/ACLQ89iZsvrAXil6H4Whp3H7JbXAPC3orv5OOxC3m05HGOrmVp8F++HX8G88MsJsxW8Wnwvc8OvZn74xUTaEl4sfoBZ4b/kk/ALiLWFPFf8ENNb3sDCsJ/Rpnofk0se418tR7EobAjx1bn8qeRJXm95E4vDBnFj2bsMqVjM6JAY9pWU8/btgxUCIE2GN4O6K7C9Vp0NnP3jjYwxY4GxAD169PCpmU5xrcmvSqR923YkRcTQsbwt+XmJdGjXjoqWMXQsa0t+fiId27XFtoyhQ2lb8vcl0jmuDSHhMbQvjSN/XyJd2rehZVgM7Q7Gkb8/kW7t2xAdFkNbT909vjWtQ2NoXdKe/AOJ9IxvRVxoDK2L25NfkEhCh1gOhsQQWxxPfkEiJ3VoRVmLSGKKOpBfmMhJHWKpbBFOVGEH8osSOaVjLNUmhMjCjuQXJZLUMQaAlgWd2Fd8uA4v6MiBkoRDddiBThSUHq5D93eisKzoUB2yvzPF5ZUkdXBqs68zBytbkBTv1HZfF0orIw/V1fndKKsuIqm9U1fmdQNbfqiuyOtGNS1IinPqsr3dqWoRQVI7py7N7U51SOtDdUludwiNJ6mtp87pQYuwLofqwpyehIR3JqmNUxfsSSA0ojNJrZ36wO4EwiI7Har32wRaRnUkqZVT76tOJCKmI0mxMRhbTf7uRKJiOpAUG0OIrSB/dyLRsfEkxcQQXh1C/p5EYmI7kBQTQ0QV5NtEWrWKJyk6huiqKvJzEmnVOp6kqBhiK8vJz02kradeVzkcSnqS2qotw/t3IaWHQgCk6TDW2uNvYMx1wKXW2ls99U3A2dbaCcd6TGpqqk1PT2/QRkVEmjNjTIa1NvVo93nzx8QdQPdadTfPx0REpBF4M6i/B5KMMYnGmHBgJPBv/7YlIiI16lyjttZWGmMmAB8DIcDr1tpVfu9MREQAL094sdb+B/iPn3sREZGj0CnkIiIBToNaRCTAaVCLiAQ4DWoRkQBX5wkvPj2pMbnAVh8f3h7Y24DtNBT1VX+B2pv6qr9A7a059dXTWht/tDv8MqhPhDEm/Vhn57hJfdVfoPamvuovUHsLlr609CEiEuA0qEVEAlwgDuqpbjdwDOqr/gK1N/VVf4HaW1D0FXBr1CIicqRA3KMWEZFaNKhFRAJcQA5qY0yyMeZbY8xSY0y6MeYst3uqYYy5yxiz1hizyhjzjNv91GaMmWiMscaYgIjVNsZM9rxXy40x7xpj2rjcT0CGNBtjuhtjFhpjVnu+r+5xu6fajDEhxpgsY8w8t3upYYxpY4yZ7fn+WmOMGex2TzWMMfd5vo4rjTEzjDERJ/qcATmogWeASdbaZOB3ntp1xpjzcfIi+1trTweedbmlQ4wx3YGLgW1u91LLJ0Bfa20/YB3wsFuNBHhIcyUw0Vp7GjAIGB9AvQHcA6xxu4kfmQJ8ZK3tA/QnQPozxnQF7gZSrbV9cS4NPfJEnzdQB7UFWnlutwZ2uthLbeOAp621ZQDW2hyX+6nteeBBnPcuIFhrF1hrKz3ltzjpQG45FNJsrS0HakKaXWet3WWtzfTcLsQZOl3d7cphjOkGXAH8w+1eahhjWgNDgX8CWGvLrbX7XW3qSKFApDEmFIiiAeZXoA7qe4HJxpjtOHutru2J/Ugv4DxjzBJjzH+NMWe63RCAMeZqYIe1dpnbvRzHr4H5Lr7+0UKaA2IY1maMSQAGAEtcbqXGCzg7ANUu91FbIpALvOFZkvmHMSba7aYArLU7cGbWNmAXcMBau+BEn9er4AB/MMZ8CnQ6yl2PAhcA91lr5xhjbsD5yXlhAPQVCrTD+fX0TGCWMeYk2wjHONbR1yM4yx6N7nh9WWvf92zzKM6v92mN2VtTY4yJAeYA91prCwKgnyuBHGtthjHm5y63U1sokALcZa1dYoyZAjwEPO5uW2CMaYvzm1oisB94xxgzxlo77YSe2FobcP+AAxw+xtsABW735OnlI+D8WvVGIN7lns4AcoAtnn+VOD/NO7n9fnn6uxlYDES53Mdg4ONa9cPAw26/P7X6CcOJu7vf7V5q9fRnnN88tgC7gRJgWgD01QnYUqs+D/jQ7b48vVwP/LNW/Svg1RN93kBd+tgJ/Mxzexiw3sVeansPOB/AGNMLCMflK3dZa1dYaztYaxOstQk4/2OlWGt3u9kXOEdZ4PzaPNxaW+JyOwEb0myMMTi/Na6x1j7ndj81rLUPW2u7eb6vRgKfW2vHuNwWnu/t7caY3p4PXQCsdrGl2rYBg4wxUZ6v6wU0wB86XVv6qMNtwBTPYnwpMNblfmq8DrxujFkJlAP/Yz0/NuWoXgZaAp8437N8a629w41GbGCHNJ8L3ASsMMYs9XzsEetklcrR3QWkeX7obgJucbkfAKyzFDMbyMT57TaLBjidXKeQi4gEuEBd+hAREQ8NahGRAKdBLSIS4DSoRUQCnAa1iEiA06AWEQlwGtQiIgHu/wODYlT39XRIsQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "x = torch.arange(-8.0, 8.0, 0.1, requires_grad=True)\n",
    "y = torch.relu(x)\n",
    "y.backward(torch.ones_like(x))\n",
    "\n",
    "plt.plot(x.detach().numpy(), y.detach().numpy(), label = 'relu')\n",
    "plt.plot(x.detach().numpy(), x.grad.numpy() ,linestyle=':', label = 'gradient')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e395766",
   "metadata": {},
   "source": [
    "**其他方法**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d01c1a39",
   "metadata": {},
   "source": [
    "决梯度消失和梯度爆炸的方法还有很多，比如下面这些：\n",
    "\n",
    "1.使用 Batch Normalization 层，这样可以缩小梯度的范围，避免梯度爆炸的问题。\n",
    "\n",
    "2.初始化权重参数，使用更加合理的初始化方法。\n",
    "\n",
    "3.使用更加稳定的优化算法，如 Adam 优化器或 RMSprop 优化器，这些优化器可以自动调整学习率，使得训练更加稳定。\n",
    "\n",
    "4.增加模型的宽度或使用残差连接，这样可以缓解深层网络中的梯度消失问题。\n",
    "\n",
    "这些方法这里暂不做深入讲解，在今后的学习中大家会一点一点的接触到的。大家加油！"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "989fd793",
   "metadata": {},
   "source": [
    "**梗直哥提示：梯度消失和梯度爆炸是深度学习中非常基础的问题，建议初学者在理解的基础上，亲自动手实践一下，相信你会记忆得更加深刻。如果你想了解更多内容，欢迎入群学习（加V: gengzhige99）**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e86d2c25",
   "metadata": {},
   "source": [
    "[Next 5-8模型文件的读写](./5-8%20模型文件的读写.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7fc7171",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
